{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Agenda\n",
      "\n",
      "- Define the problem and the approach\n",
      "- <p style=\"color: red\">Data basics: loading data, looking at your data, basic commands</p>\n",
      "- Handling missing values\n",
      "- Intro to scikit-learn\n",
      "- Grouping and aggregating data\n",
      "- Feature selection\n",
      "- Fitting and evaluating a model\n",
      "- Deploying your work"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##In this notebook you will\n",
      "\n",
      "- Learn how to load data into Python\n",
      "- Learn the basics of working with data in `pandas`\n",
      "- Clean and manage your data\n",
      "- Wrangle missing data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Reading from a file"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "import pylab as pl\n",
      "import numpy as np\n",
      "import re"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We're going to use the <code>read_csv</code> function in pandas"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "?pd.read_csv"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.read_csv(\"./data/credit-training.csv\", index_col=0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##What is <code>df</code>?\n",
      "Our data is represented by a DataFrame. You can think of data frames as a giant spreadsheet which you can program. It's a collection of series (or columns) with a common set of commands that make managing data in Python super easy."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#barf!!!\n",
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 150000 entries, 1 to 150000\n",
        "Data columns:\n",
        "SeriousDlqin2yrs                        150000  non-null values\n",
        "RevolvingUtilizationOfUnsecuredLines    150000  non-null values\n",
        "age                                     150000  non-null values\n",
        "NumberOfTime30-59DaysPastDueNotWorse    150000  non-null values\n",
        "DebtRatio                               150000  non-null values\n",
        "MonthlyIncome                           120269  non-null values\n",
        "NumberOfOpenCreditLinesAndLoans         150000  non-null values\n",
        "NumberOfTimes90DaysLate                 150000  non-null values\n",
        "NumberRealEstateLoansOrLines            150000  non-null values\n",
        "NumberOfTime60-89DaysPastDueNotWorse    150000  non-null values\n",
        "NumberOfDependents                      146076  non-null values\n",
        "dtypes: float64(4), int64(7)"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#you'll learn more about head() later\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SeriousDlqin2yrs</th>\n",
        "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
        "      <th>age</th>\n",
        "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
        "      <th>DebtRatio</th>\n",
        "      <th>MonthlyIncome</th>\n",
        "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
        "      <th>NumberOfTimes90DaysLate</th>\n",
        "      <th>NumberRealEstateLoansOrLines</th>\n",
        "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
        "      <th>NumberOfDependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.766127</td>\n",
        "      <td> 45</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.802982</td>\n",
        "      <td>  9120</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.957151</td>\n",
        "      <td> 40</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.121876</td>\n",
        "      <td>  2600</td>\n",
        "      <td>  4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.658180</td>\n",
        "      <td> 38</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.085113</td>\n",
        "      <td>  3042</td>\n",
        "      <td>  2</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.233810</td>\n",
        "      <td> 30</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.036050</td>\n",
        "      <td>  3300</td>\n",
        "      <td>  5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.907239</td>\n",
        "      <td> 49</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.024926</td>\n",
        "      <td> 63588</td>\n",
        "      <td>  7</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "   SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines  age  \\\n",
        "1                 1                              0.766127   45   \n",
        "2                 0                              0.957151   40   \n",
        "3                 0                              0.658180   38   \n",
        "4                 0                              0.233810   30   \n",
        "5                 0                              0.907239   49   \n",
        "\n",
        "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
        "1                                     2   0.802982           9120   \n",
        "2                                     0   0.121876           2600   \n",
        "3                                     1   0.085113           3042   \n",
        "4                                     0   0.036050           3300   \n",
        "5                                     1   0.024926          63588   \n",
        "\n",
        "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
        "1                               13                        0   \n",
        "2                                4                        0   \n",
        "3                                2                        1   \n",
        "4                                5                        0   \n",
        "5                                7                        0   \n",
        "\n",
        "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
        "1                             6                                     0   \n",
        "2                             0                                     0   \n",
        "3                             0                                     0   \n",
        "4                             0                                     0   \n",
        "5                             1                                     0   \n",
        "\n",
        "   NumberOfDependents  \n",
        "1                   2  \n",
        "2                   1  \n",
        "3                   0  \n",
        "4                   0  \n",
        "5                   0  "
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(df)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "pandas.core.frame.DataFrame"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can think of a Series is a column in your data. A series will have a particular datatype associated with it. Datatypes can be integers, strings, floating point numbers, etc."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print df.SeriousDlqin2yrs.head()\n",
      "type(df.SeriousDlqin2yrs)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1    1\n",
        "2    0\n",
        "3    0\n",
        "4    0\n",
        "5    0\n",
        "Name: SeriousDlqin2yrs\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "pandas.core.series.Series"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.dtypes"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "SeriousDlqin2yrs                          int64\n",
        "RevolvingUtilizationOfUnsecuredLines    float64\n",
        "age                                       int64\n",
        "NumberOfTime30-59DaysPastDueNotWorse      int64\n",
        "DebtRatio                               float64\n",
        "MonthlyIncome                           float64\n",
        "NumberOfOpenCreditLinesAndLoans           int64\n",
        "NumberOfTimes90DaysLate                   int64\n",
        "NumberRealEstateLoansOrLines              int64\n",
        "NumberOfTime60-89DaysPastDueNotWorse      int64\n",
        "NumberOfDependents                      float64"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Basic Commands\n",
      "Many of these commands will be familiar for those of you coming from a UNIX/Linux background. If not, don't sweat it, these commands are super easy, but extremely helpful for \"taking a look around\" at your data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Selecting a column"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df['DebtRatio']\n",
      "df.DebtRatio"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "1        0.802982\n",
        "2        0.121876\n",
        "3        0.085113\n",
        "4        0.036050\n",
        "5        0.024926\n",
        "6        0.375607\n",
        "7     5710.000000\n",
        "8        0.209940\n",
        "9       46.000000\n",
        "10       0.606291\n",
        "11       0.309476\n",
        "12       0.531529\n",
        "13       0.298354\n",
        "14       0.382965\n",
        "15     477.000000\n",
        "...\n",
        "149986       0.324962\n",
        "149987       0.080384\n",
        "149988       0.055692\n",
        "149989       0.347924\n",
        "149990       0.001408\n",
        "149991       0.609779\n",
        "149992       0.477658\n",
        "149993    4132.000000\n",
        "149994       0.000000\n",
        "149995       0.404293\n",
        "149996       0.225131\n",
        "149997       0.716562\n",
        "149998    3870.000000\n",
        "149999       0.000000\n",
        "150000       0.249908\n",
        "Name: DebtRatio, Length: 150000"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###head(n=5)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SeriousDlqin2yrs</th>\n",
        "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
        "      <th>age</th>\n",
        "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
        "      <th>DebtRatio</th>\n",
        "      <th>MonthlyIncome</th>\n",
        "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
        "      <th>NumberOfTimes90DaysLate</th>\n",
        "      <th>NumberRealEstateLoansOrLines</th>\n",
        "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
        "      <th>NumberOfDependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.766127</td>\n",
        "      <td> 45</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.802982</td>\n",
        "      <td>  9120</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.957151</td>\n",
        "      <td> 40</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.121876</td>\n",
        "      <td>  2600</td>\n",
        "      <td>  4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.658180</td>\n",
        "      <td> 38</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.085113</td>\n",
        "      <td>  3042</td>\n",
        "      <td>  2</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.233810</td>\n",
        "      <td> 30</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.036050</td>\n",
        "      <td>  3300</td>\n",
        "      <td>  5</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.907239</td>\n",
        "      <td> 49</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.024926</td>\n",
        "      <td> 63588</td>\n",
        "      <td>  7</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "   SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines  age  \\\n",
        "1                 1                              0.766127   45   \n",
        "2                 0                              0.957151   40   \n",
        "3                 0                              0.658180   38   \n",
        "4                 0                              0.233810   30   \n",
        "5                 0                              0.907239   49   \n",
        "\n",
        "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
        "1                                     2   0.802982           9120   \n",
        "2                                     0   0.121876           2600   \n",
        "3                                     1   0.085113           3042   \n",
        "4                                     0   0.036050           3300   \n",
        "5                                     1   0.024926          63588   \n",
        "\n",
        "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
        "1                               13                        0   \n",
        "2                                4                        0   \n",
        "3                                2                        1   \n",
        "4                                5                        0   \n",
        "5                                7                        0   \n",
        "\n",
        "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
        "1                             6                                     0   \n",
        "2                             0                                     0   \n",
        "3                             0                                     0   \n",
        "4                             0                                     0   \n",
        "5                             1                                     0   \n",
        "\n",
        "   NumberOfDependents  \n",
        "1                   2  \n",
        "2                   1  \n",
        "3                   0  \n",
        "4                   0  \n",
        "5                   0  "
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.head(1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SeriousDlqin2yrs</th>\n",
        "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
        "      <th>age</th>\n",
        "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
        "      <th>DebtRatio</th>\n",
        "      <th>MonthlyIncome</th>\n",
        "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
        "      <th>NumberOfTimes90DaysLate</th>\n",
        "      <th>NumberRealEstateLoansOrLines</th>\n",
        "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
        "      <th>NumberOfDependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.766127</td>\n",
        "      <td> 45</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.802982</td>\n",
        "      <td> 9120</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "   SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines  age  \\\n",
        "1                 1                              0.766127   45   \n",
        "\n",
        "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
        "1                                     2   0.802982           9120   \n",
        "\n",
        "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
        "1                               13                        0   \n",
        "\n",
        "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
        "1                             6                                     0   \n",
        "\n",
        "   NumberOfDependents  \n",
        "1                   2  "
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.SeriousDlqin2yrs.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "1    1\n",
        "2    0\n",
        "3    0\n",
        "4    0\n",
        "5    0\n",
        "Name: SeriousDlqin2yrs"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###tail(n=5)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SeriousDlqin2yrs</th>\n",
        "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
        "      <th>age</th>\n",
        "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
        "      <th>DebtRatio</th>\n",
        "      <th>MonthlyIncome</th>\n",
        "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
        "      <th>NumberOfTimes90DaysLate</th>\n",
        "      <th>NumberRealEstateLoansOrLines</th>\n",
        "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
        "      <th>NumberOfDependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>149996</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.040674</td>\n",
        "      <td> 74</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.225131</td>\n",
        "      <td> 2100</td>\n",
        "      <td>  4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>149997</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.299745</td>\n",
        "      <td> 44</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.716562</td>\n",
        "      <td> 5584</td>\n",
        "      <td>  4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>149998</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.246044</td>\n",
        "      <td> 58</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3870.000000</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> 18</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>149999</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 30</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.000000</td>\n",
        "      <td> 5716</td>\n",
        "      <td>  4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>150000</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.850283</td>\n",
        "      <td> 64</td>\n",
        "      <td> 0</td>\n",
        "      <td>    0.249908</td>\n",
        "      <td> 8158</td>\n",
        "      <td>  8</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "        SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines  age  \\\n",
        "149996                 0                              0.040674   74   \n",
        "149997                 0                              0.299745   44   \n",
        "149998                 0                              0.246044   58   \n",
        "149999                 0                              0.000000   30   \n",
        "150000                 0                              0.850283   64   \n",
        "\n",
        "        NumberOfTime30-59DaysPastDueNotWorse    DebtRatio  MonthlyIncome  \\\n",
        "149996                                     0     0.225131           2100   \n",
        "149997                                     0     0.716562           5584   \n",
        "149998                                     0  3870.000000            NaN   \n",
        "149999                                     0     0.000000           5716   \n",
        "150000                                     0     0.249908           8158   \n",
        "\n",
        "        NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
        "149996                                4                        0   \n",
        "149997                                4                        0   \n",
        "149998                               18                        0   \n",
        "149999                                4                        0   \n",
        "150000                                8                        0   \n",
        "\n",
        "        NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
        "149996                             1                                     0   \n",
        "149997                             1                                     0   \n",
        "149998                             1                                     0   \n",
        "149999                             0                                     0   \n",
        "150000                             2                                     0   \n",
        "\n",
        "        NumberOfDependents  \n",
        "149996                   0  \n",
        "149997                   2  \n",
        "149998                   0  \n",
        "149999                   0  \n",
        "150000                   0  "
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.RevolvingUtilizationOfUnsecuredLines.tail()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "149996    0.040674\n",
        "149997    0.299745\n",
        "149998    0.246044\n",
        "149999    0.000000\n",
        "150000    0.850283\n",
        "Name: RevolvingUtilizationOfUnsecuredLines"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###describe(percentile_width=50)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>SeriousDlqin2yrs</th>\n",
        "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
        "      <th>age</th>\n",
        "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
        "      <th>DebtRatio</th>\n",
        "      <th>MonthlyIncome</th>\n",
        "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
        "      <th>NumberOfTimes90DaysLate</th>\n",
        "      <th>NumberRealEstateLoansOrLines</th>\n",
        "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
        "      <th>NumberOfDependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td>  120269.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 150000.000000</td>\n",
        "      <td> 146076.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td>      0.066840</td>\n",
        "      <td>      6.048438</td>\n",
        "      <td>     52.295207</td>\n",
        "      <td>      0.421033</td>\n",
        "      <td>    353.005076</td>\n",
        "      <td>    6670.221237</td>\n",
        "      <td>      8.452760</td>\n",
        "      <td>      0.265973</td>\n",
        "      <td>      1.018240</td>\n",
        "      <td>      0.240387</td>\n",
        "      <td>      0.757222</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>      0.249746</td>\n",
        "      <td>    249.755371</td>\n",
        "      <td>     14.771866</td>\n",
        "      <td>      4.192781</td>\n",
        "      <td>   2037.818523</td>\n",
        "      <td>   14384.674215</td>\n",
        "      <td>      5.145951</td>\n",
        "      <td>      4.169304</td>\n",
        "      <td>      1.129771</td>\n",
        "      <td>      4.155179</td>\n",
        "      <td>      1.115086</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.029867</td>\n",
        "      <td>     41.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.175074</td>\n",
        "      <td>    3400.000000</td>\n",
        "      <td>      5.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.154181</td>\n",
        "      <td>     52.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.366508</td>\n",
        "      <td>    5400.000000</td>\n",
        "      <td>      8.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.559046</td>\n",
        "      <td>     63.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.868254</td>\n",
        "      <td>    8249.000000</td>\n",
        "      <td>     11.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      2.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td>      1.000000</td>\n",
        "      <td>  50708.000000</td>\n",
        "      <td>    109.000000</td>\n",
        "      <td>     98.000000</td>\n",
        "      <td> 329664.000000</td>\n",
        "      <td> 3008750.000000</td>\n",
        "      <td>     58.000000</td>\n",
        "      <td>     98.000000</td>\n",
        "      <td>     54.000000</td>\n",
        "      <td>     98.000000</td>\n",
        "      <td>     20.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "       SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines            age  \\\n",
        "count     150000.000000                         150000.000000  150000.000000   \n",
        "mean           0.066840                              6.048438      52.295207   \n",
        "std            0.249746                            249.755371      14.771866   \n",
        "min            0.000000                              0.000000       0.000000   \n",
        "25%            0.000000                              0.029867      41.000000   \n",
        "50%            0.000000                              0.154181      52.000000   \n",
        "75%            0.000000                              0.559046      63.000000   \n",
        "max            1.000000                          50708.000000     109.000000   \n",
        "\n",
        "       NumberOfTime30-59DaysPastDueNotWorse      DebtRatio   MonthlyIncome  \\\n",
        "count                         150000.000000  150000.000000   120269.000000   \n",
        "mean                               0.421033     353.005076     6670.221237   \n",
        "std                                4.192781    2037.818523    14384.674215   \n",
        "min                                0.000000       0.000000        0.000000   \n",
        "25%                                0.000000       0.175074     3400.000000   \n",
        "50%                                0.000000       0.366508     5400.000000   \n",
        "75%                                0.000000       0.868254     8249.000000   \n",
        "max                               98.000000  329664.000000  3008750.000000   \n",
        "\n",
        "       NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
        "count                    150000.000000            150000.000000   \n",
        "mean                          8.452760                 0.265973   \n",
        "std                           5.145951                 4.169304   \n",
        "min                           0.000000                 0.000000   \n",
        "25%                           5.000000                 0.000000   \n",
        "50%                           8.000000                 0.000000   \n",
        "75%                          11.000000                 0.000000   \n",
        "max                          58.000000                98.000000   \n",
        "\n",
        "       NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
        "count                 150000.000000                         150000.000000   \n",
        "mean                       1.018240                              0.240387   \n",
        "std                        1.129771                              4.155179   \n",
        "min                        0.000000                              0.000000   \n",
        "25%                        0.000000                              0.000000   \n",
        "50%                        1.000000                              0.000000   \n",
        "75%                        2.000000                              0.000000   \n",
        "max                       54.000000                             98.000000   \n",
        "\n",
        "       NumberOfDependents  \n",
        "count       146076.000000  \n",
        "mean             0.757222  \n",
        "std              1.115086  \n",
        "min              0.000000  \n",
        "25%              0.000000  \n",
        "50%              0.000000  \n",
        "75%              1.000000  \n",
        "max             20.000000  "
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.age.describe(percentile_width=25)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "count    150000.000000\n",
        "mean         52.295207\n",
        "std          14.771866\n",
        "min           0.000000\n",
        "37.5%        47.000000\n",
        "50%          52.000000\n",
        "62.5%        57.000000\n",
        "max         109.000000"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###unqiue() and nunique()"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.MonthlyIncome.unique()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "array([  9120.,   2600.,   3042., ...,  15648.,    521.,   7647.])"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.MonthlyIncome.nunique()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "13594"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###pd.value_counts(values_to_count)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pd.value_counts(df.NumberOfDependents)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "0     86902\n",
        "1     26316\n",
        "2     19522\n",
        "3      9483\n",
        "4      2862\n",
        "5       746\n",
        "6       158\n",
        "7        51\n",
        "8        24\n",
        "10        5\n",
        "9         5\n",
        "20        1\n",
        "13        1"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pd.value_counts(df.NumberOfDependents, ascending=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "13        1\n",
        "20        1\n",
        "9         5\n",
        "10        5\n",
        "8        24\n",
        "7        51\n",
        "6       158\n",
        "5       746\n",
        "4      2862\n",
        "3      9483\n",
        "2     19522\n",
        "1     26316\n",
        "0     86902"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pd.value_counts(df.NumberOfDependents, sort=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "0     86902\n",
        "1     26316\n",
        "2     19522\n",
        "3      9483\n",
        "4      2862\n",
        "5       746\n",
        "6       158\n",
        "7        51\n",
        "8        24\n",
        "9         5\n",
        "10        5\n",
        "13        1\n",
        "20        1"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#chain value_counts together with head() to give you the top 3\n",
      "pd.value_counts(df.NumberOfDependents).head(3)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "0    86902\n",
        "1    26316\n",
        "2    19522"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##pd.crosstab(rows, cols)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pd.crosstab(df.number_of_times90_days_late, df.serious_dlqin2yrs)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "AttributeError",
       "evalue": "'DataFrame' object has no attribute 'number_of_times90_days_late'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-21-618a6354e31d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcrosstab\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumber_of_times90_days_late\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserious_dlqin2yrs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.pyc\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m   2044\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2045\u001b[0m         raise AttributeError(\"'%s' object has no attribute '%s'\" %\n\u001b[0;32m-> 2046\u001b[0;31m                              (type(self).__name__, name))\n\u001b[0m\u001b[1;32m   2047\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2048\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mAttributeError\u001b[0m: 'DataFrame' object has no attribute 'number_of_times90_days_late'"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####Use `pd.crosstab` to make a table that contains customer's ages in the lefthand column and the number of dependents they have in the right"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "table = pd.crosstab(..., ...)\n",
      "assert table.shape==(84, 13)\n",
      "table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "SyntaxError",
       "evalue": "invalid syntax (<ipython-input-22-ab646e07fa71>, line 1)",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-22-ab646e07fa71>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    table = pd.crosstab(..., ...)\u001b[0m\n\u001b[0m                        ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Basic Cleanliness"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's fix for formatting of the column names. I personally like snake_case (and so does Python). I found [this handy function](http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-camel-case) on stackoverflow for converting camelCase to snake_case.\n",
      "\n",
      "Now we can apply the camel_to_snake function on each column name."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load https://gist.github.com/glamp/6529725/raw/e38ffd2fc4cb840be21098486ffe5df991946736/camel_to_snake.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def camel_to_snake(column_name):\n",
      "    \"\"\"\n",
      "    converts a string that is camelCase into snake_case\n",
      "    Example:\n",
      "        print camel_to_snake(\"javaLovesCamelCase\")\n",
      "        > java_loves_camel_case\n",
      "    See Also:\n",
      "        http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-camel-case\n",
      "    \"\"\"\n",
      "    s1 = re.sub('(.)([A-Z][a-z]+)', r'\\1_\\2', column_name)\n",
      "    return re.sub('([a-z0-9])([A-Z])', r'\\1_\\2', s1).lower()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "camel_to_snake(\"javaLovesCamelCase\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "'java_loves_camel_case'"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.columns = [camel_to_snake(col) for col in df.columns]\n",
      "df.columns.tolist()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "['serious_dlqin2yrs',\n",
        " 'revolving_utilization_of_unsecured_lines',\n",
        " 'age',\n",
        " 'number_of_time30-59_days_past_due_not_worse',\n",
        " 'debt_ratio',\n",
        " 'monthly_income',\n",
        " 'number_of_open_credit_lines_and_loans',\n",
        " 'number_of_times90_days_late',\n",
        " 'number_real_estate_loans_or_lines',\n",
        " 'number_of_time60-89_days_past_due_not_worse',\n",
        " 'number_of_dependents']"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Slicing and Indexing Data\n",
      "pandas (like R) uses a system of boolean indexing. What this means is that when selecting particular rows or columns in your dataset..."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Grabbing columns"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df['monthly_income'].head()\n",
      "df.monthly_income.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 21,
       "text": [
        "1     9120\n",
        "2     2600\n",
        "3     3042\n",
        "4     3300\n",
        "5    63588\n",
        "Name: monthly_income"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[['monthly_income', 'serious_dlqin2yrs']].head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>  9120</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>  2600</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td>  3042</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td>  3300</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 63588</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 22,
       "text": [
        "   monthly_income  serious_dlqin2yrs\n",
        "1            9120                  1\n",
        "2            2600                  0\n",
        "3            3042                  0\n",
        "4            3300                  0\n",
        "5           63588                  0"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "columns_i_want = ['monthly_income', 'serious_dlqin2yrs']\n",
      "df[columns_i_want].head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>  9120</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>  2600</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td>  3042</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td>  3300</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 63588</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 23,
       "text": [
        "   monthly_income  serious_dlqin2yrs\n",
        "1            9120                  1\n",
        "2            2600                  0\n",
        "3            3042                  0\n",
        "4            3300                  0\n",
        "5           63588                  0"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Creating New Columns\n",
      "You can use dot notation to access existing columns, but you cannot use dot notation to create new columns. The following ```df.newcolumn = 1``` actually sets a new property to the `df` object."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.newcolumn = 1\n",
      "# this will throw an error\n",
      "df['newcolumn']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "KeyError",
       "evalue": "u'no item named newcolumn'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-24-320c417e7351>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnewcolumn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# this will throw an error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'newcolumn'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.pyc\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   1969\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Cannot index using non-boolean DataFrame'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1970\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1971\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_item_cache\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1972\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1973\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_getitem_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/generic.pyc\u001b[0m in \u001b[0;36m_get_item_cache\u001b[0;34m(self, item)\u001b[0m\n\u001b[1;32m    532\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 534\u001b[0;31m             \u001b[0mvalues\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    535\u001b[0m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_box_item_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    536\u001b[0m             \u001b[0mcache\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/internals.pyc\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, item)\u001b[0m\n\u001b[1;32m    882\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    883\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 884\u001b[0;31m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mblock\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_find_block\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    885\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mblock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    886\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/internals.pyc\u001b[0m in \u001b[0;36m_find_block\u001b[0;34m(self, item)\u001b[0m\n\u001b[1;32m   1014\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1015\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_find_block\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1016\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_have\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1017\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mblock\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblocks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1018\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mitem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mblock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/internals.pyc\u001b[0m in \u001b[0;36m_check_have\u001b[0;34m(self, item)\u001b[0m\n\u001b[1;32m   1021\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_check_have\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1022\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mitem\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1023\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'no item named %s'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpprint_thing\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1024\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1025\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mreindex_axis\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnew_axis\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mKeyError\u001b[0m: u'no item named newcolumn'"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df['one'] = 1\n",
      "df.one.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "1    1\n",
        "2    1\n",
        "3    1\n",
        "4    1\n",
        "5    1\n",
        "Name: one"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Removing a column"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "del df['one']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Boolean Indexing"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.monthly_income > 5000"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 27,
       "text": [
        "1      True\n",
        "2     False\n",
        "3     False\n",
        "4     False\n",
        "5      True\n",
        "6     False\n",
        "7     False\n",
        "8     False\n",
        "9     False\n",
        "10     True\n",
        "11    False\n",
        "12     True\n",
        "13     True\n",
        "14     True\n",
        "15    False\n",
        "...\n",
        "149986    False\n",
        "149987    False\n",
        "149988    False\n",
        "149989     True\n",
        "149990     True\n",
        "149991    False\n",
        "149992     True\n",
        "149993    False\n",
        "149994    False\n",
        "149995    False\n",
        "149996    False\n",
        "149997     True\n",
        "149998    False\n",
        "149999     True\n",
        "150000     True\n",
        "Name: monthly_income, Length: 150000"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "n_gt_5k = len(df[df.monthly_income > 5000])\n",
      "print n_gt_5k,\"people with monthly_income > 5000\"\n",
      "df[df.monthly_income > 5000].head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "64410 people with monthly_income > 5000\n"
       ]
      },
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <th>age</th>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <th>debt_ratio</th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <th>number_of_dependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.766127</td>\n",
        "      <td> 45</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.802982</td>\n",
        "      <td>  9120</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.907239</td>\n",
        "      <td> 49</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.024926</td>\n",
        "      <td> 63588</td>\n",
        "      <td>  7</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.189169</td>\n",
        "      <td> 57</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.606291</td>\n",
        "      <td> 23684</td>\n",
        "      <td>  9</td>\n",
        "      <td> 0</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.018798</td>\n",
        "      <td> 51</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.531529</td>\n",
        "      <td>  6501</td>\n",
        "      <td>  7</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.010352</td>\n",
        "      <td> 46</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.298354</td>\n",
        "      <td> 12454</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "    serious_dlqin2yrs  revolving_utilization_of_unsecured_lines  age  \\\n",
        "1                   1                                  0.766127   45   \n",
        "5                   0                                  0.907239   49   \n",
        "10                  0                                  0.189169   57   \n",
        "12                  0                                  0.018798   51   \n",
        "13                  0                                  0.010352   46   \n",
        "\n",
        "    number_of_time30-59_days_past_due_not_worse  debt_ratio  monthly_income  \\\n",
        "1                                             2    0.802982            9120   \n",
        "5                                             1    0.024926           63588   \n",
        "10                                            0    0.606291           23684   \n",
        "12                                            0    0.531529            6501   \n",
        "13                                            0    0.298354           12454   \n",
        "\n",
        "    number_of_open_credit_lines_and_loans  number_of_times90_days_late  \\\n",
        "1                                      13                            0   \n",
        "5                                       7                            0   \n",
        "10                                      9                            0   \n",
        "12                                      7                            0   \n",
        "13                                     13                            0   \n",
        "\n",
        "    number_real_estate_loans_or_lines  \\\n",
        "1                                   6   \n",
        "5                                   1   \n",
        "10                                  4   \n",
        "12                                  2   \n",
        "13                                  2   \n",
        "\n",
        "    number_of_time60-89_days_past_due_not_worse  number_of_dependents  \n",
        "1                                             0                     2  \n",
        "5                                             0                     0  \n",
        "10                                            0                     2  \n",
        "12                                            0                     2  \n",
        "13                                            0                     2  "
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.ix[40:45, 0:2]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>40</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.304491</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>41</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.719985</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>42</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.072898</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>43</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.469057</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>44</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.718705</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>45</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.368876</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "    serious_dlqin2yrs  revolving_utilization_of_unsecured_lines\n",
        "40                  0                                  0.304491\n",
        "41                  0                                  0.719985\n",
        "42                  0                                  0.072898\n",
        "43                  0                                  0.469057\n",
        "44                  0                                  0.718705\n",
        "45                  0                                  0.368876"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mask = (df.monthly_income > 5000) & (df.serious_dlqin2yrs==1)\n",
      "df[mask].head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <th>age</th>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <th>debt_ratio</th>\n",
        "      <th>monthly_income</th>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <th>number_of_dependents</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1  </th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.766127</td>\n",
        "      <td> 45</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.802982</td>\n",
        "      <td>  9120</td>\n",
        "      <td> 13</td>\n",
        "      <td> 0</td>\n",
        "      <td> 6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14 </th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.964673</td>\n",
        "      <td> 40</td>\n",
        "      <td> 3</td>\n",
        "      <td> 0.382965</td>\n",
        "      <td> 13700</td>\n",
        "      <td>  9</td>\n",
        "      <td> 3</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75 </th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.133063</td>\n",
        "      <td> 49</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.182882</td>\n",
        "      <td> 10257</td>\n",
        "      <td>  9</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>163</th>\n",
        "      <td> 1</td>\n",
        "      <td> 1.046279</td>\n",
        "      <td> 47</td>\n",
        "      <td> 1</td>\n",
        "      <td> 1.104301</td>\n",
        "      <td>  5416</td>\n",
        "      <td>  6</td>\n",
        "      <td> 0</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>185</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.085338</td>\n",
        "      <td> 51</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.477504</td>\n",
        "      <td>  6200</td>\n",
        "      <td> 11</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "     serious_dlqin2yrs  revolving_utilization_of_unsecured_lines  age  \\\n",
        "1                    1                                  0.766127   45   \n",
        "14                   1                                  0.964673   40   \n",
        "75                   1                                  0.133063   49   \n",
        "163                  1                                  1.046279   47   \n",
        "185                  1                                  0.085338   51   \n",
        "\n",
        "     number_of_time30-59_days_past_due_not_worse  debt_ratio  monthly_income  \\\n",
        "1                                              2    0.802982            9120   \n",
        "14                                             3    0.382965           13700   \n",
        "75                                             1    0.182882           10257   \n",
        "163                                            1    1.104301            5416   \n",
        "185                                            0    0.477504            6200   \n",
        "\n",
        "     number_of_open_credit_lines_and_loans  number_of_times90_days_late  \\\n",
        "1                                       13                            0   \n",
        "14                                       9                            3   \n",
        "75                                       9                            0   \n",
        "163                                      6                            0   \n",
        "185                                     11                            0   \n",
        "\n",
        "     number_real_estate_loans_or_lines  \\\n",
        "1                                    6   \n",
        "14                                   1   \n",
        "75                                   2   \n",
        "163                                  2   \n",
        "185                                  1   \n",
        "\n",
        "     number_of_time60-89_days_past_due_not_worse  number_of_dependents  \n",
        "1                                              0                     2  \n",
        "14                                             1                     2  \n",
        "75                                             0                     3  \n",
        "163                                            0                     1  \n",
        "185                                            0                     0  "
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Write some code that filters to the following:\n",
      "\n",
      "- people 35 or older\n",
      "- who have not been delinquent in the past 2 years\n",
      "- who have less than 10 open credit lines/loans"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#DELETE#mask = (df.age >= 35) & (df.serious_dlqin2yrs==0) & (df.number_of_open_credit_lines_and_loans < 10)\n",
      "mask = ...\n",
      "len(df[mask])==76151"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 41,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####Write something that filters:\n",
      "\n",
      "- people who have been delinquent in the past 2 years\n",
      "- have are in the 90th percentile for monthly_income\n",
      "\n",
      "*HINT: use `quantile`*"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#DELETE#mask = (df.serious_dlqin2yrs==1) & \\\n",
      "#        (df.monthly_income >= df.monthly_income.quantile(.9))\n",
      "mask = ...\n",
      "len(df[mask])==539"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 44,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 44
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Handling Null Values\n",
      "One of the most frustrating parts of data science can be handling null or missing data. pandas has a lot of built in features for making is super easy to handle missing data. The first thing we need to do is determine which fields have missing data. To do that we're going to use `pd.melt`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###[Long vs. Wide Data](http://en.wikipedia.org/wiki/Wide_and_narrow_data)\n",
      "Depending on the problem you're solving, you may need to rotate between having your data in wide/long format.\n",
      "\n",
      "Wide data is probably what you think of when the work \"spreadsheet\" comes to mind. We're talking about data in which each row represents 1 datapoint and each value is in a particular column. This is well suited for things like modeling and producing summary statistics.\n",
      "\n",
      "I often find that having data in `long` format is often best for doing the same task against multiple variables. Things like plotting distributions of each variable, making frequency tables, or, in our case, determining what portion of a dataframe's variables are null."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###pd.melt()\n",
      "For converting data from `wide` to `long` format.\n",
      "```\n",
      ">>> df\n",
      "A B C\n",
      "a 1 2\n",
      "b 3 4\n",
      "c 5 6\n",
      "\n",
      ">>> pd.melt(df, id_vars=['A'], value_vars=['B'])\n",
      "A variable value\n",
      "a B        1\n",
      "b B        3\n",
      "c B        5\n",
      "```"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "?pd.melt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 64
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# By not specifying id_vars, we're going to melt EVERYTHING\n",
      "df_lng = pd.melt(df)\n",
      "# now our data is a series of (key, value) rows. \n",
      "#think of when you've done this in Excel so that you can\n",
      "#create a pivot table \n",
      "df_lng.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>variable</th>\n",
        "      <th>value</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> serious_dlqin2yrs</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> serious_dlqin2yrs</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> serious_dlqin2yrs</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> serious_dlqin2yrs</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> serious_dlqin2yrs</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 45,
       "text": [
        "            variable  value\n",
        "0  serious_dlqin2yrs      1\n",
        "1  serious_dlqin2yrs      0\n",
        "2  serious_dlqin2yrs      0\n",
        "3  serious_dlqin2yrs      0\n",
        "4  serious_dlqin2yrs      0"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "null_variables = df_lng.value.isnull()\n",
      "null_variables.sum()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 46,
       "text": [
        "33655"
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# crosstab creates a frequency table between 2 variables\n",
      "# it's going to automatically enumerate the possibilities between\n",
      "# the two Series and show you a count of occurrences \n",
      "#in each possible bucket\n",
      "pd.crosstab(df_lng.variable, null_variables)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th>value</th>\n",
        "      <th>False</th>\n",
        "      <th>True</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>variable</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>age</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>debt_ratio</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>monthly_income</th>\n",
        "      <td> 120269</td>\n",
        "      <td> 29731</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_dependents</th>\n",
        "      <td> 146076</td>\n",
        "      <td>  3924</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 47,
       "text": [
        "value                                         False  True \n",
        "variable                                                  \n",
        "age                                          150000      0\n",
        "debt_ratio                                   150000      0\n",
        "monthly_income                               120269  29731\n",
        "number_of_dependents                         146076   3924\n",
        "number_of_open_credit_lines_and_loans        150000      0\n",
        "number_of_time30-59_days_past_due_not_worse  150000      0\n",
        "number_of_time60-89_days_past_due_not_worse  150000      0\n",
        "number_of_times90_days_late                  150000      0\n",
        "number_real_estate_loans_or_lines            150000      0\n",
        "revolving_utilization_of_unsecured_lines     150000      0\n",
        "serious_dlqin2yrs                            150000      0"
       ]
      }
     ],
     "prompt_number": 47
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# let's abstract that code into a function so we can easily \n",
      "# recalculate it\n",
      "def print_null_freq(df):\n",
      "    \"\"\"\n",
      "    for a given DataFrame, calculates how many values for \n",
      "    each variable is null and prints the resulting table to stdout\n",
      "    \"\"\"\n",
      "    df_lng = pd.melt(df)\n",
      "    null_variables = df_lng.value.isnull()\n",
      "    return pd.crosstab(df_lng.variable, null_variables)\n",
      "print_null_freq(df)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th>value</th>\n",
        "      <th>False</th>\n",
        "      <th>True</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>variable</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>age</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>debt_ratio</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>monthly_income</th>\n",
        "      <td> 120269</td>\n",
        "      <td> 29731</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_dependents</th>\n",
        "      <td> 146076</td>\n",
        "      <td>  3924</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 48,
       "text": [
        "value                                         False  True \n",
        "variable                                                  \n",
        "age                                          150000      0\n",
        "debt_ratio                                   150000      0\n",
        "monthly_income                               120269  29731\n",
        "number_of_dependents                         146076   3924\n",
        "number_of_open_credit_lines_and_loans        150000      0\n",
        "number_of_time30-59_days_past_due_not_worse  150000      0\n",
        "number_of_time60-89_days_past_due_not_worse  150000      0\n",
        "number_of_times90_days_late                  150000      0\n",
        "number_real_estate_loans_or_lines            150000      0\n",
        "revolving_utilization_of_unsecured_lines     150000      0\n",
        "serious_dlqin2yrs                            150000      0"
       ]
      }
     ],
     "prompt_number": 48
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####Use pd.melt to create a data frame in the following format:\n",
      "```\n",
      "     serious_dlqin2yrs variable\t  value\n",
      "0\t                1\t age\t    45\n",
      "1\t                0\t age\t    40\n",
      "2\t                0\t age\t    38\n",
      "3\t                0\t age\t    30\n",
      "4\t                0\t age\t    49\n",
      "...\t                ...\t ...\t    ...\n",
      "299999              1\t debt_ratio 0.423\n",
      "300000              0\t debt_ratio 0.8923\n",
      "```\n",
      "Only include values for `age` and `debt_ratio`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#DELETE#melted = pd.melt(df, id_vars=[\"serious_dlqin2yrs\"], value_vars=[\"age\", \"debt_ratio\"])\n",
      "melted = pd.melt(..., id_vars=[...], value_vars=[...])\n",
      "\n",
      "print len(melted)==300000\n",
      "print melted.variable.unique()==np.array(['age', 'debt_ratio'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "True\n",
        "[ True  True]\n"
       ]
      }
     ],
     "prompt_number": 50
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Filling NA's"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s = pd.Series([1, 2, None, 4])\n",
      "s"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 51,
       "text": [
        "0     1\n",
        "1     2\n",
        "2   NaN\n",
        "3     4"
       ]
      }
     ],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s.fillna(3)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 52,
       "text": [
        "0    1\n",
        "1    2\n",
        "2    3\n",
        "3    4"
       ]
      }
     ],
     "prompt_number": 52
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s.ffill()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 53,
       "text": [
        "0    1\n",
        "1    2\n",
        "2    2\n",
        "3    4"
       ]
      }
     ],
     "prompt_number": 53
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s.bfill()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 54,
       "text": [
        "0    1\n",
        "1    2\n",
        "2    4\n",
        "3    4"
       ]
      }
     ],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s.fillna(s.mean())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 55,
       "text": [
        "0    1.000000\n",
        "1    2.000000\n",
        "2    2.333333\n",
        "3    4.000000"
       ]
      }
     ],
     "prompt_number": 55
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you look at `df` you can see that there are 2 columns which don't have a full 150,000 values: `monthly_income` and `number_of_dependents`. In order to incorporate these variables into our analysis, we need to specify how to treat these missing values."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For number_of_dependents let's keep things simple and intuitive. if someone didn't specify how many dependents they had then let's assume it's becasue they don't have any to begin with."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.number_of_dependents = df.number_of_dependents.fillna(0)\n",
      "# proof that the number_of_dependents no longer contains nulls\n",
      "print_null_freq(df)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th>value</th>\n",
        "      <th>False</th>\n",
        "      <th>True</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>variable</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>age</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>debt_ratio</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>monthly_income</th>\n",
        "      <td> 120269</td>\n",
        "      <td> 29731</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_dependents</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_open_credit_lines_and_loans</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time30-59_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_time60-89_days_past_due_not_worse</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_of_times90_days_late</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>number_real_estate_loans_or_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>revolving_utilization_of_unsecured_lines</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>serious_dlqin2yrs</th>\n",
        "      <td> 150000</td>\n",
        "      <td>     0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 56,
       "text": [
        "value                                         False  True \n",
        "variable                                                  \n",
        "age                                          150000      0\n",
        "debt_ratio                                   150000      0\n",
        "monthly_income                               120269  29731\n",
        "number_of_dependents                         150000      0\n",
        "number_of_open_credit_lines_and_loans        150000      0\n",
        "number_of_time30-59_days_past_due_not_worse  150000      0\n",
        "number_of_time60-89_days_past_due_not_worse  150000      0\n",
        "number_of_times90_days_late                  150000      0\n",
        "number_real_estate_loans_or_lines            150000      0\n",
        "revolving_utilization_of_unsecured_lines     150000      0\n",
        "serious_dlqin2yrs                            150000      0"
       ]
      }
     ],
     "prompt_number": 56
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Taking a look at `monthly_income` we see that it's a bit more complicated than `number_of_dependents`. We have a few options for replacing missing data. We could do something like set it to the mean or median or the dataset but this might skew our distribution. We could also set it to 0 but this might not be right either. Instead we're going to use a technique called imputation. We'll go into this more after we take a look at `scikit-learn`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.monthly_income.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 57,
       "text": [
        "count     120269.000000\n",
        "mean        6670.221237\n",
        "std        14384.674215\n",
        "min            0.000000\n",
        "25%         3400.000000\n",
        "50%         5400.000000\n",
        "75%         8249.000000\n",
        "max      3008750.000000"
       ]
      }
     ],
     "prompt_number": 57
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.to_csv(\"./data/credit-data-post-import.csv\", index=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 58
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##We just did the following\n",
      "\n",
      "- Used `read_csv` to load data Python\n",
      "- Used `head`, `tail`, `unique`, and other functions to inspect our data\n",
      "- Sliced data and selected data frames\n",
      "- Cleaned our data and learned the basics of handling missing values"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}